{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "774fc28c-6e02-46f4-9441-991764d6bad1",
   "metadata": {},
   "source": [
    "# 数据迭代\n",
    "我们继续学习MindSpore中的数据处理操作，在一节中，我们介绍如何进行数据迭代。为了提高模型的精度，我们需要数据迭代操作，即重复喂给网络模型相同的数据。\n",
    "\n",
    "在MindSpore中，进行数据迭代有两种操作：\n",
    "- 直接调用高阶API中的Model接口\n",
    "- 使用迭代器进行迭代数据\n",
    "\n",
    "使用Model接口进行模型训练就已经包含了数据迭代的操作，这里我们着重了解使用迭代器来进行数据迭代。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce82db99-5f88-4e51-8278-db9665e04178",
   "metadata": {},
   "source": [
    "## 创建迭代器\n",
    "在MindSpore中有两种迭代器，分别为：\n",
    "- `元组迭代器`：使用`create_tuple_iterator`创建，通常在Model.train内部使用。\n",
    "- `字典迭代器`：使用`create_dict_iterator`创建，通常用于自定义训练过程时。用户可以根据字典的`key`对具体数据进行进一步的处理。\n",
    "\n",
    "迭代后数据类型为mindspore.Tensor类型，下面简单介绍一下两种迭代器的使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "af694928-0484-4bee-85f3-964be98ff5f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] ME(33580:33112,MainProcess):2022-10-23-15:49:34.686.084 [mindspore\\dataset\\engine\\datasets_user_defined.py:656] Python multiprocessing is not supported on Windows platform.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " create tuple iterator\n",
      "item:\n",
      " [[1 2]\n",
      " [3 4]]\n",
      "item:\n",
      " [[5 6]\n",
      " [7 8]]\n",
      "\n",
      " create dict iterator\n",
      "item:\n",
      " [[1 2]\n",
      " [3 4]]\n",
      "item:\n",
      " [[5 6]\n",
      " [7 8]]\n"
     ]
    }
   ],
   "source": [
    "import mindspore.dataset as ds\n",
    "# 创建数据集\n",
    "dataset = [[[1, 2], [3, 4]], [[5, 6], [7, 8]]]\n",
    "\n",
    "# 加载数据集\n",
    "dataset = ds.NumpySlicesDataset(dataset, column_names=[\"data\"], shuffle=False)\n",
    "\n",
    "# 创建元组迭代器\n",
    "print(\"\\n create tuple iterator\")\n",
    "for item in dataset.create_tuple_iterator():\n",
    "    print(\"item:\\n\", item[0])\n",
    "\n",
    "# 创建字典迭代器\n",
    "print(\"\\n create dict iterator\")\n",
    "for item in dataset.create_dict_iterator():\n",
    "    print(\"item:\\n\", item['data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae94cf8d-ddce-4f45-b9e1-d215bc91fd8e",
   "metadata": {},
   "source": [
    "我们可以看到两种迭代器主要的差别在于如何提取数据，字典迭代器需要使用先前指定的`\"data\"`标签做索引，元组迭代器则直接索引下标0。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d50a0f8-25ba-4344-b11d-f3e583cf28c8",
   "metadata": {
    "tags": []
   },
   "source": [
    "## epoch参数\n",
    "epoch值表示指定迭代的次数，使用方法为在`create_dict_iterator`或`create_tuple_iterator`命令中指定参数`num_epochs`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e80b504a-833b-4499-8b12-0bc43527ae61",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:  0\n",
      "item: \n",
      " [[1 2]\n",
      " [3 4]]\n",
      "item: \n",
      " [[5 6]\n",
      " [7 8]]\n",
      "epoch:  1\n",
      "item: \n",
      " [[1 2]\n",
      " [3 4]]\n",
      "item: \n",
      " [[5 6]\n",
      " [7 8]]\n"
     ]
    }
   ],
   "source": [
    "epoch = 2\n",
    "dataset = dataset.create_dict_iterator(num_epochs=epoch)\n",
    "for i in range(epoch):\n",
    "    print(\"epoch: \", i)\n",
    "    for item in dataset:\n",
    "        print(\"item: \\n\", item[\"data\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27b9b796",
   "metadata": {},
   "source": [
    "其中还有一个`output_numpy`参数，表示是否设置输出数据类型为`numpy.ndarray`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4d05a391",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[WARNING] ME(4876:13644,MainProcess):2022-10-22-18:05:40.860.758 [mindspore\\dataset\\engine\\datasets_user_defined.py:656] Python multiprocessing is not supported on Windows platform.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n",
      "<class 'numpy.ndarray'>\n",
      "<class 'mindspore.common.tensor.Tensor'>\n",
      "<class 'mindspore.common.tensor.Tensor'>\n"
     ]
    }
   ],
   "source": [
    "dataset = [[[1, 2], [3, 4]], [[5, 6], [7, 8]]]\n",
    "dataset = ds.NumpySlicesDataset(dataset, column_names=[\"data\"], shuffle=False)\n",
    "for item in dataset.create_dict_iterator(output_numpy=True):\n",
    "    print(type(item['data']))\n",
    "for item in dataset.create_dict_iterator(output_numpy=False):\n",
    "    print(type(item['data']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f39e9fbb",
   "metadata": {},
   "source": [
    "## 迭代输入数据\n",
    "关于如何向神经网络迭代输入数据的内容所涉及的知识有很多我们还没有学到，故只在此展示部分代码，如下图：\n",
    "![](./在网络中迭代输入数据.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mindspore",
   "language": "python",
   "name": "mindvision"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
